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COMMAND  AND  GENERAL  STAFF  OFFICER  COURSE  EVALUATION 
OF  THE  TACTICAL  PLANNING  WORKSTATION 


Introduction 

The  Tactical  Planning  Workstation  is  an  automated  system  to  support  Army 
tactical  staff  performance  at  the  division  level.  It  was  developed  within  the 
Experimental  Development,  Demonstration,  and  Integration  Center  at  the  Army 
Research  Institute  Ft.  Leavenworth  Field  Unit  as  a  platform  for  prototyping  staff 
information  and  decision  aids  and  conducting  research  on  staff  performance. 
See  Flanagan  and  Fallesen  (in  preparation)  for  a  description  of  the  capabilities 
of  the  system  and  Packard,  McClanahan,  Zarse,  and  Ross  (in  preparation)  for  a 
software  description.  This  paper  documents  an  evaluation  of  the  Workstation 
that  was  conducted  to  obtain  preliminary  feedback  from  potential  users  on  key 
design  features  of  the  Workstation.  Students  at  the  Command  and  General 
S'aff  Officer  Course  (CGSOC)  at  Ft.  Leavenworth  provided  data  for  the 
evaluation. 


Background 

The  students  made  their  evaluation  assessments  as  part  of  their  Command 
and  General  Staff  Officer  Course,  Advanced  Command  and  Control  Elective, 
A399.  This  30  hour  elective  was  developed  to  provide  an  instructional 
environment  for  students  to  learn  the  capabilities  of  the  Maneuver  Control 
System  (MCS);  to  determine  the  requirements,  potentials,  and  shortfalls  of 
integrating  MCS  into  the  CGSOC  curriculum;  and  to  review  and  evaluate  the 
MCS  software  and  provide  feedback  to  the  combat  development  community. 
The  demonstration  they  received  of  the  Tactical  Planning  Workstation  was  part 
of  a  three  hour  block  of  instruction  devoted  to  observing  demonstrations  of 
computer  systems  other  than  MCS.  The  Tactical  Planning  Workstation  served 
as  a  comparison  system  to  MCS.  The  instructors  compiled  the  student  MCS 
assessments  into  a  report  on  problems  and  recommendations  (subject:  After 
action  report,  Advanced  C2  Elective;  dated  2  Jun  89)  that  was  provided  to  the 
MCS  TRADOC  System  Manager  (TSM)  for  resolution.  For  ARI,  the  student 
comments  about  the  Workstation  and  completed  questionnaires  provided  user 
feedback  for  an  evaluation  of  the  Workstation  interface. 

Evaluation  Method 

During  17  to  18  May  1989,  eighteen  Command  and  General  Staff  College 
students  were  given  a  half  hour  demonstration  of  the  Tactical  Planning 
Workstation  and  an  opportunity  to  use  it.  They  were  then  asked  to  respond  to  a 
questionnaire  on  the  Workstation  and  asked  for  verbal  comments.  Eight  people 
filled  out  the  questionnaire.  This  paper  summarizes  students'  verbal  comments 
and  questionnaire  responses. 

Not  all  of  the  students  observing  the  demonstrations  completed  a 
questionnaire.  Eight  of  the  18  students  did  fill  one  out.  The  questionnaires 
were  completed  on  a  volunteer  basis.  Most  of  those  not  complying  with  the 
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request  to  complete  the  questionnaire  made  verbal  comments  about  the 
interface.  The  non-respondents  to  the  questionnaire  may  have  seen  no 
additional  value  in  completing  it;  felt  they  did  not  fully  understand  the 
capabilities  sufficiently  to  rate  them;  not  wanted  to  take  the  time  to  complete  the 
questionnaire  before  their  next  class;  or  have  not  been  favorably  impressed  and 
so  made  no  written  comment. 

Results  of  the  Questionnaire 

Appendix  A  shows  the  questionnaire  items  and  the  Mean,  Median,  and 
Standard  Deviation  for  each  applicable  item. 

The  students  generally  rated  the  Workstation  very  easy  to  use.  The  situation 
data  and  reference  data  menus,  walking  menus,  map  features  and  task 
organization  all  received  a  mean  rating  on  ease  of  use  of  4.8  on  a  scale  of  1 
(hard  to  use)  to  5  (easy  to  use). 

Of  the  map  features,  the  grids  and  contours,  respectively,  were  rated  first  and 
second  most  useful  by  all  but  one  student.  For  map  background  features, 
shaded  relief,  vegetation,  and  elevation  handling  were  rated  equally  useful.  On 
the  map  scales,  1 :80,000,  1 :1 60,  000,  1 :400,000,  and  1 :800,000  were  ranked 
first  through  fourth  in  terms  of  usefulness.  One  student  commented  that  the 
usefulness  of  the  map  scales  depends  on  the  planning  level  (e.g.,  division  or 
corp).  The  ratings  of  the  usefulness  of  the  various  tactical  overlay  features 
varied  considerably,  "Show  BLUEFOR  Units"  was  rated  most  useful,  and  "Show 
OPFOR  Control  Measures”  was  rated  the  least  useful. 

Verbal  Comments 

One  interviewer  was  present  to  record  comments  and  the  following 
represents  only  a  sample  of  the  verbal  comments  of  the  students. 

Comparison  to  the  Maneuver  Control  System  (MCS^ 

Students  were  very  favorably  impressed  with  the  ease  of  use  of  the 
Workstation,  especially  when  compared  to  MCS.  Typical  comments  were 
"Compared  to  MCS,  this  is  Christmas!"  and  "An  absolutely  outstanding 
interface.  We  must  incorporate  this  into  MCS."  One  student  commented  that  he 
liked  to  be  in  control  of  what's  displayed  on  the  screen.  In  MCS  the  system 
controls  what  is  shown.  One  student  thought  that  the  busy  light  was  a  useful 
feature  -  in  MCS  it  was  difficult  to  tell  if  the  computer  is  processing  material. 

Ease  of  Learning 

The  students  thought  the  Workstation  was  very  easy  to  learn.  However,  in 
the  half  hour  instruction  time,  not  all  the  features  were  shown,  and  the  students 
might  have  found  features  not  shown  harder  to  learn.  Students  thought  it  was 
important  to  have  an  interface  that  could  be  self  taught  rather  than  requiring  90 
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hours  to  learn  (a  reference  to  the  duration  of  the  program  of  instruction  for 
MCS).  The  students  felt  that  officers  have  enough  to  do  without  having  to  learn 
a  complicated  skill  and  then  maintain  it,  especially  a  very  perishable  one.  One 
student  said  he  was  often  frustrated  with  MCS  because  he  could  not  remember 
the  correct  commands. 

Walking  Menus 

Several  students  commented  that  it  would  be  more  natural  to  click  on  a 
selection  rather  than  having  to  hold  the  mouse  button  down.  Some  students 
had  a  tendency  to  let  up  on  the  mouse  button  and  have  the  menu  disappear 
inadvertently.  Several  thought  the  walking  menu  would  be  hard  to  use  under 
conditions  of  stress  and  fatigue  because  of  the  fine  motor  coordination  required. 

These  verbal  comments  do  not  seem  to  be  consistent  with  students' 
questionnaire  ratings  of  the  walking  menu  as  "easy  to  use".  (See  question  7  in 
Appendix  A).  It  may  be  that  students  thought  the  walking  menu  was  easy  to 
use,  but  that  their  suggestions  would  make  it  even  easier  to  use.  Another 
explanation  for  the  discrepancy  is  that  the  questionnaire  item  may  be 
ambiguous  to  the  students.  The  item  asks  "How  easy  was  it  to  use  the  walking 
menus  to  request  displays  and  perform  actions?"  This  could  be  perceived  as 
two  different  questions:  (1)  How  easy  was  it  cognitively  to  request  displays? 
and  (2)  How  easy  was  it  physically  to  perform  the  actions?  Students  could  have 
been  responding  to  the  first  part  of  the  question.  In  any  case,  it  appears  that 
students  thought  the  walking  menu  was  not  difficult  to  use,  but  that  it  could  be 
improved. 

Scrolling 

Students  general’y  liked  the  scroll  technique  and  thought  it  was  better  than 
the  one  on  MCS.  Some  had  a  hard  time  getting  in  the  correct  scroll  band.  One 
student  suggested  that  using  keys  to  scroll  would  be  easier  than  using  the 
mouse.  esDecially  under  conditions  of  fatigue  and  time  stress  when  "the  motor 
coordination  is  the  first  thing  to  go." 

Map  Scales 

Several  students  said  the  map  scales  should  be  standard  ones  rather  than 
the  non-standard  scales  available  on  the  Workstation.  They  thought  otherwise 
they  would  not  be  able  to  print  graphic  overlays  to  fit  standard  maps.  However, 
they  did  not  realize  that  a  map  printed  by  the  system  could  be  scaled  differently 
than  the  scale  of  the  display. 

Information  ReloeyaJ 

Students  thought  the  Workstation  allowed  easier  access  to  information  than 
MCS.  The  walking  menus  did  not  reauire  students  to  remember  what  was  in 
the  data  base.  One  said  this  was  important  because  soldiers  under  pressure 
should  not  be  required  to  remember  detailed  instructions.  However,  several 
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commented  that  the  organization  of  the  data  base  (Flanagan  and  Fallesen,  in 
preparation)  by  FM  numbers  would  be  more  natural  and  easy  to  use  than  the 
current  system. 

Mouse  Buttons 

There  was  some  confusion  about  when  to  use  each  of  the  three  mouse 
buttons. 

Task  Organization  Tool  and  Status  Tool  (TOASTS 

TOAST  is  a  staff  aid  which  graphically  displays  the  current  task  organization 
of  friendly  units.  Using  the  mouse,  users  can  graphically  change  the  task 
organization  or  obtain  detailed  or  primary  status  information  on  any  unit. 
Students  thought  these  features  would  be  especially  useful  in  tactical  planning. 

Suggestions  for  Improvement 

One  student  suggested  the  addition  of  an  automatic  warning  if  unit  strength 
fell  below  70%.  In  general,  he  felt  the  system  should  be  designed  using  a 
"management  by  exception"  orientation  to  planning.  Another  said  that  the  term 
"Build"  to  label  the  Build  window  did  not  mean  anything  to  him.  A  more 
descriptive  term  should  be  used.  One  student  suggested  that  drawing  control 
measures  should  be  done  with  a  continuous  line,  (the  same  as  some  Macintosh 
drawing  capabilities),  not  point  to  point. 

Summary 

Students  made  favorable  comments  about  the  ease  of  use  and  ease  of 
learning  of  the  Workstation,  especially  when  compared  with  the  MCS  interface. 
Students  especially  liked  the  minimization  of  memory  requirements  for 
Workstation  operation.  They  showed  some  confusion  about  when  to  select  from 
the  right  mouse  button  or  the  left,  and  they  thought  the  walking  menu  might  be 
hard  to  use  under  conditions  of  fatigue,  stress  and  MOPP  4.  Suggestions  for 
improvement  included  using  keys  to  scroll  a  window  and  clicking  on  menu 
selections  rather  than  having  to  hold  the  mouse  button  down. 
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APPENDIX  A 

Interface  Questionnaire  Showing  Means,  Medians,  and  Standard  Deviations  (SD) 

of  Questionnaire  Scores 

Name: _ 


HUMAN-MACHINE  INTERFACE  EVALUATION  (EDDIC) 

Below  are  a  series  of  questions  concerning  your  impressions  of 
using  the  Sun  workstation  for  COA  analysis. 

1 .  How  well  are  the  Situation  Data  and  Reference  Data  Menus  labeled  and  organized  for  easy 
and  logical  access  to  data  (please  circle  the  appropriate  number  in  the  rating  scale  below)? 

12345  Mean  =  4.8 

_  Median  =  5 

Hard  to  use  OK  Easy  to  use  SD  =  .4 

2  For  the  purpose  of  conducting  a  COA  analysis,  are  the  map  background,  map  scale,  map 
features,  and  tactical  overlay  laid  out  in  an  operationally  logical  manner  so  you  can  find  the 
information  you  want  quickly  (please  circle  the  appropriate  number  in  the  rating  scale 
below)? 

12345  Mean  =  4.8 

_ _  Median  =  5 

Hard  to  use  OK  Easy  to  use  SD  =  .4 

Please  specify  the  areas  you  feel  need  improvement.  If  possible,  please  suggest 
improvements. 


3  In  each  of  the  map  display  categories  below,  please  rank  order  the  individual  attributes  from 
one  (1)  for  the  most  useful  attribute,  to  four  (4)  for  the  least  useful  attribute  (1  -  6  for  the 
Map  Features  category). 


SD 

Med  ian 

Mean 

MAP  FEATURES 

SD 

Median 

Mean 

MAP  SCALES 

.4 

1 

1  .  1 

Grids 

.4 

1 

1.2 

1  :  80,000 

.  4 

') 

1.9 

Contours 

.5 

2 

1.8 

1  :  160,000 

t  .0 

3 

3 

Roads 

.5 

3 

3.2 

1  :  400,000 

1.2 

4 

3.9 

Hydrography 

.5 

4 

3.8 

1  :  800,000 

1 .  1 

4 

4.1 

Built-up  Areas 

0 

6 

6 

Misc.  Features 

TACTICAL  OVERLAY 

SD 

Median 

SD 

Median 

MAP  BACKGROUND 

1.0 

1 

1.6 

Show  BLUEFOR  Units 

.9 

2 

1.9 

Show  BLUEFOR  Control 

1  .  1 

3 

2.  1 

Shaded  Relief 

Measures 

.  6 

z. 

2.0 

Vegetation 

1.3 

2 

2.4 

Show  OPFOh  Units 

.7 

2 

1  .  7 

Elevation  Banding 

1.3 

2 

2.7 

Show  OPFOR  Control 

0 

4 

4.0 

None 

Measures 

A- 1 


4.  Are  the  labels  (or  the  map  display  categories  in  question  3  above  consistent  with 
terminology  used  by  Army  personnel? 

YES  NO  All  said  yes 

It  NO,  please  circle  some  of  the  worst  offending  labels  in  the  list  in  question  3. 


5  While  identifying  Critical  Events,  Was  it  easy  to  enter  CEs  in  the  tactical  overlay  using  the 
Sun  workstation? 

YES  NO  N/’A 

If  NO.  explain  the  problem. 


6.  How  easy  was  it  to  coordinate  the  use  of  both  the  Sun  and  Symbolics  workstations  while 
identifying  Critical  Events  (please  circle  the  appropriate  number  in  the  rating  scale  below)? 

1  2  3  4  5 

_ N/A 

Difficult  Somewhat  Somewhat  Easy  Very 

Difficult  Easy  Easy 


If  rated  (1)  or  (2),  explain  the  problem. 


7.  How  easy  was  it  to  use  the  “walking"  menus  to  request  displays  and  perform  actions  (please 
circle  the  appropriate  number  in  the  rating  scale  below)? 

1  2  3  4  5  Me an  =4. 8 

_ : _ _  Median  =  5 

Hard  to  use  OK  Easy  to  use  sd  =  .5 


8.  How  easy  was  it  to  use  the  Task  Organization  tool  to  review  and  modify  the  task 

organization,  and  obtain  updated  unit  status  (p'ease  circle  the  appropriate  number  in  the 
rating  scale  below)? 

12345  Mean  =  4.8 

_  Median  =  5 

Hard  to  use  OK  Easy  to  use  sd  =  .5 
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